Method and system for the detection and augmentation of tactile interactions in augmented reality

ABSTRACT

Program actions may be initiated after detection of a predefined gesture by a user with a real-world object. Users may interact with their physical environment in an augmented reality by detecting interactions with real objects using a combination of location and motion detection, material identification using wearable sensors, or both. Based on detected sensor data from user interaction with a real-world object, a predefined gesture may be identified and a program action associated with that target interaction for the real-world object may be executed. In some cases, the user experience may be enhanced by providing haptic feedback in response to tactile gestures and resulting events.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/493,678, filed Sep. 12, 2019, which is a national stageapplication under 35 U.S.C. 371 of International Application No.PCT/US2018/022976, filed on Mar. 16, 2018, which claims benefit under 35U.S.C. § 119(e) from U.S. Provisional Patent Application Ser. No.62/474,341, filed Mar. 21, 2017, all of which are incorporated herein byreference in their entirety.

BACKGROUND

Recently, AR games such as Niantic's Pokemon Go have enjoyed success.Interactions with AR applications currently take place through atouchscreen (e.g., flicking a ball to a virtual creature on atouchscreen) or through movements captured by GPS (e.g., walking onekilometer to release a virtual creature). Emerging interactiontechniques also include free-air gestures (e.g., Microsoft Hololens) andwand-mediated interactions (e.g., Oculus Touch, Razer Hydra, ViveController).

Interactions with mobile AR applications are limited and do not allowrich, direct interactions with the user's environment. Morespecifically, these applications do not take advantage of the tactileinteractions that are possible in a user's surroundings. Althoughengaging, these types of tactile interactions are not supported bycurrent AR applications in part because no practical solution exists todetect them.

Systems relying on free-air gestures and wand-like controllers can beused to detect gestures but have not been used to detect rich tactileinteractions with a real environment. They also require additionalhardware beyond a smartphone and smart watch or activity tracker, and,in the case of the wand, require holding a device that limitsmanipulations of the environment.

Discussion of haptics and object interactions may be found in US PatentPublications 2016/0179198 and 2016/0179199. The use of sensors attachedto a probe, including vibration sensors, to detect the properties of thesurfaces being touched is discussed in Romano and Kuchenbecker, “Methodsfor robotic tool-mediated haptic surface recognition,” 2014 IEEE HapticsSymposium, pp. 49-56, and Strese, et al., “Multimodal Feature-basedSurface Material Classification”, IEEE Transactions on Haptics, pp. 1,5555.

The systems and methods disclosed herein address the above issues, andothers.

SUMMARY

In one embodiment, a method may comprise: operating a mobile device of auser to determine the user's location; based on the determined locationof the user, retrieving information identifying (i) at least a firstnearby real-world object, (ii) at least a first predefined gestureassociated with the first real-world object, (ii) at least a firstprogram action associated with the first predefined gesture, and (iv) atleast a first haptic feedback response associated with the first programaction; operating at least one sensor in communication with the mobiledevice to detect movement of the user; and responsive to a determinationthat the detected movement of the user matches the first predefinedgesture: initiating the first program action; and controlling at least afirst haptic feedback component in communication with the mobile deviceto execute the first haptic feedback response.

In one embodiment, a method may comprise: determining a user's locationby at least one sensor of a mobile device of the user; retrieving, basedon the determined location of the user, a list of nearby real-worldobjects each having at least one associated grouping of a targetinteraction, a program action, and a haptic response; detecting, with atleast one sensor of the mobile device, a target interaction performed bythe user; and responsive to matching the detected target interactionperformed by the user to a first nearby real-world object of theretrieved list, initiating the program action and haptic responseassociated with the first nearby real-world object. In some cases, thetarget interaction may comprise a tactile interaction, a gesture action,or a combination thereof.

In one embodiment, a method comprises determining a location of a firstuser with at least one sensor of a mobile device; retrieving at least afirst object entry from an object database based on the determinedlocation of the first user, the first object entry comprising anidentifier of a first proximate object, at least one gesture actionassociated with the first proximate object, and at least one programaction associated with each of the gesture actions; detecting a firstgesture performed by the first user with at least one sensor of themobile device; matching the detected first gesture to at least onegesture action in the first object entry; and executing the at least oneprogram action associated with the matched at least one gesture action.

In one embodiment, a method comprises initiating detection mode of amobile device to await detecting of a tactile interaction of a firstuser with a first object; detecting movements and vibrations caused bythe first user interacting with the first object in a naturalenvironment, with at least one sensor of the mobile device; andresponsive to a determination that detected movements and vibrationsindicate a particular tactile interaction has occurred, initiating afirst program action associated with said tactile interaction in amemory of the mobile device.

In some embodiments, a system may comprise a processor and anon-transitory storage medium storing instructions operative, whenexecuted on the processor, to perform functions including those setforth above, and others.

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed understanding may be had from the following description,presented by way of example in conjunction with the accompanyingdrawings in which like reference numerals in the figures indicate likeelements, and wherein:

FIG. 1 illustrates components of a system for location-based detectionof tactile interactions with the environment, according to anembodiment.

FIG. 2 illustrates a flow chart for location-based detection of tactileinteractions with the environment, according to an embodiment.

FIG. 3 illustrates components of a system for sensor-based detection oftactile interactions with the environment, according to an embodiment.

FIG. 4 illustrates a flow chart for sensor-based detection of tactileinteractions with the environment, according to an embodiment.

FIGS. 5A-5C illustrate an exemplary embodiment of an AR user experience.

FIG. 6 illustrates a flow diagram for an exemplary embodiment set forthherein.

FIG. 7 illustrates an exemplary wireless transmit/receive unit (WTRU)that may be employed as a mobile device in some embodiments.

FIG. 8 illustrates an exemplary network entity that may be employed insome embodiments.

DETAILED DESCRIPTION

A detailed description of illustrative embodiments will now be providedwith reference to the various Figures. Although this descriptionprovides detailed examples of possible implementations, it should benoted that the provided details are intended to be by way of example andin no way limit the scope of the application.

Note that various hardware elements of one or more of the describedembodiments are referred to as “modules” that carry out (i.e., perform,execute, and the like) various functions that are described herein inconnection with the respective modules. As used herein, a moduleincludes hardware (e.g., one or more processors, one or moremicroprocessors, one or more microcontrollers, one or more microchips,one or more application-specific integrated circuits (ASICs), one ormore field programmable gate arrays (FPGAs), one or more memory devices)deemed suitable by those of skill in the relevant art for a givenimplementation. Each described module may also include instructionsexecutable for carrying out the one or more functions described as beingcarried out by the respective module, and it is noted that thoseinstructions could take the form of or include hardware (i.e.,hardwired) instructions, firmware instructions, software instructions,and/or the like, and may be stored in any suitable non-transitorycomputer-readable medium or media, such as commonly referred to as RAM,ROM, etc.

In various embodiments, different approaches may be used to detecttactile interactions with the environment. In some cases, multipleapproaches may be used together to improve the quality of the results.

Types of objects that may be interacted with include, but are notlimited to, the following.

Outdoors interactions may include, for example and without limitation,interactions with nature, street furniture, the ground, buildings,visual content, parked cars, people, etc. Nature, for example, mayinclude the trunk, branches or leaves of a tree or bush; a flower; aplant; a rock; sand; grass; water (e.g., a lake, a river, a puddle);snow; ice; etc. Street furniture, for example, may include a park bench,a lamp post, a stop sign, a traffic light, an orange cone, a mailbox, afire hydrant, a memorial, a statue, a fountain, a guardrail, a garbagebin, a bus shelter, etc. The ground, for example, may include a gravelpath, grass, a curb cut, textured pavement for the visually impaired,the edge of a sidewalk, etc. Buildings, for example, may include a wall,a door, a door handle, a window, a sign, a mat or carpet at the entranceof a building, etc. Visual content, for example, may include anadvertisement (e.g., a poster, a video display), a sign, graffiti, etc.Parked cars, for example, may include a personal car, a shared car(e.g., car2go), a self-driving car (e.g., autonomous or automatedvehicle, or semi-autonomous or semi-automated vehicle), a taxi or carproviding a ride service (e.g., Lyft or Uber), etc. Interactions with aperson may include, for example, shaking hands with someone, or tappingon someone's back, or the like.

Indoors interactions may include, for example and without limitation,architectural elements, furniture, the floor, electrical devices, movingparts, others, etc. Architectural elements may include, for example, awall, a door, a door handle, a window, etc. Furniture may include, forexample, a table, a counter, a chair, a sofa, an armrest, etc. The floormay include, for example, tiles, carpet, etc. Electrical devices mayinclude, for example, a lamp, a television, speakers, a vending machine,etc. Moving parts may include, for example, a switch, a slider, abutton, a latch, or the like. Others may include, for example, a patternon a wall (or a painting, tapestry, etc.), clothes, etc.

Location-Based Approach. In one embodiment, location detection may becombined with gesture sensing in order to detect tactile interactionswith a user's environment. First, location services (e.g., geolocation,GPS, indoor positioning such as with beacons, compass, etc.) may be usedto determine the location (or position relative to a fixed point) of auser and query a database for nearby objects of interest. The user'slocation may include the user's orientation relative to their localenvironment (e.g., what is the user looking at). Gesture sensing (e.g.,an accelerometer in a smartwatch or fitness tracker, a wearable motionsensor, external optical sensors) may then detect motion of the user'shand (or other body part). An algorithm may determine the likelihoodthat one or more recognized tactile gestures have been performed againstan object of interest. For example, repetitive left-right motion on avertical plane may trigger an action when performed in the proximity ofa wall, as if resulting from brushing against the wall.

Sensor-Based Approach. In one embodiment, a sensor may be used to detectone or more material properties of an object being touched. For example,an accelerometer of a smartwatch or fitness tracker may be used todetect vibrations produced by an interaction with an object. Analgorithm may determine the properties of a material being touched basedon the detected vibration pattern. Motion estimation, such as from anaccelerometer, may be used in some embodiments to improve theclassification.

In some embodiments, these and other approaches may further be augmentedby producing haptic feedback in response to tactile gestures andresulting events in an application. The haptic feedback may, forexample, be used to confirm that a gesture is being recognized (e.g., avibration of increasing intensity indicating progress towards a goal) orthat an event has been triggered (e.g., a popping vibration). The hapticfeedback could furthermore pertain to virtual objects, making the ARtactile. For example, the feel of a virtual creature could be simulatedthrough vibrations in a watch when brushing against a wall.

The systems and methods disclosed herein may enable tactile interactionswith a user's environment. Adding a tactile aspect to AR games may makethe games more enjoyable and engaging, and provide more diversegameplay. It may also provide an educational benefit by encouragingplayers to explore their environment with touch. The augmentation ofthese tactile interactions with haptic feedback may also improve theirusability and the realism of the augmented reality.

The methods and systems described herein do not call for anyinstrumentation of the environment, or holding a controller. The methodsand systems may operate in some embodiments with only a smartphone (orAR headset, etc.) and a wristband such as a smart watch or fitnesstracker.

Systems and methods described herein may utilize sensor or hapticcomponents of one or more devices or systems associated with the user.For instance, a user may have as one mobile device a smartphone (whichmay, for instance, have a sensor for determining the user's location),and also be wearing a smartwatch which may have a vibration sensor and ahaptic feedback component. The various sensors, components, and devicesmay be in communication with each other, such that the smartphone mayreceive sensor data and control a haptic feedback component.

In one embodiment, a method may comprise: operating a mobile device of auser to determine the user's location; based on the determined locationof the user, retrieving information identifying (i) at least a firstnearby real-world object, (ii) at least a first predefined gestureassociated with the first real-world object, (ii) at least a firstprogram action associated with the first predefined gesture, and (iv) atleast a first haptic feedback response associated with the first programaction; operating at least one sensor in communication with the mobiledevice to detect movement of the user; responsive to a determinationthat the detected movement of the user matches the first predefinedgesture: initiating the first program action; and controlling at least afirst haptic feedback component in communication with the mobile deviceto execute the first haptic feedback response.

In some instances, the retrieved information may also comprise alocation of the first real-world object, and determining whether thedetected movement matches the first predefined gesture may includedetermining whether the user is touching the first real-world objectassociated with the first predefined gesture by: determining whether theuser is within a threshold distance of the first real-world object basedon the location of the first real-world object and the user's location;and determining, based on vibration detected by at least one sensor incommunication with the mobile device, whether the user is touching anobject.

In some instances, the retrieved information may further comprise avibration profile of the first predefined gesture, and determiningwhether the detected movement matches the first predefined gesture mayinclude determining whether the detected movement of the user matchesthe vibration profile of the first predefined gesture by: determining,based on vibration detected by at least one sensor in communication withthe mobile device, that the user is touching an object; evaluating withat least one classification algorithm a likelihood of the at least firstpredefined gesture based on each of the detected movement and thedetected vibration; calculating an average of the separate likelihoods;and selecting the first predefined gesture if the average of thelikelihoods exceeds a threshold value. In some cases, the retrievedinformation may further comprise a location of the first real-worldobject, and determining whether the detected movement matches the firstpredefined gesture may further include: determining whether the user iswithin a threshold distance of the first real-world object based on thelocation of the first real-world object and the user's location.

In some instances, the retrieved information may further comprise avibration profile of the first predefined gesture associated with thefirst real-world object, and determining whether the detected movementmatches the first predefined gesture may comprise: determining, based onvibration detected by at least one sensor in communication with themobile device, that the user is touching an object; evaluating with atleast one classification algorithm a likelihood of the first predefinedgesture based on the detected vibration; and matching the detectedmovement of the user to the vibration profile of the first predefinedgesture if the likelihood of the first predefined gesture exceeds athreshold.

In some instances, the retrieved information may further comprise alocation of the first real-world object and a vibration profileassociated with the first predefined gesture and the first real-worldobject, and determining whether the detected movement matches the firstpredefined gesture may include determining whether the user is touchingthe first real-world object associated with the first predefined gestureby: determining whether the user is within a threshold distance of thefirst real-world object based on the location of the first real-worldobject and the user's location; determining, based on vibration detectedby at least one sensor in communication with the mobile device, that theuser is touching an object; and determining that the detected vibrationmatches the retrieved vibration profile.

In some instances, the retrieved information may further comprise alocation of the first real-world object, and determining whether thedetected movement matches the first predefined gesture may include:determining that the user is within a threshold distance of the firstreal-world object based on the location of the first real-world objectand the user's location; evaluating with at least one classificationalgorithm a likelihood of the first predefined gesture based on thedetected movement; and matching the detected movement to the firstpredefined gesture if the likelihood exceeds a threshold value.

In some instances, the retrieved information may further comprise avibration profile of the first predefined gesture, said vibrationprofile associated with a first texture of a surface of the firstreal-world object, and determining whether the detected movement matchesthe first predefined gesture may comprise: determining, based onvibration detected by at least one sensor in communication with themobile device, that the user is touching an object; evaluating with atleast one classification algorithm a likelihood of the first predefinedgesture based on the detected vibration; and matching the detectedvibration to the retrieved vibration profile of the first predefinedgesture.

In some instances, matching the detected movement of the user to thefirst predefined gesture may comprise: evaluating with at least oneclassification algorithm a likelihood of each of the at least firstpredefined gesture based on the detected movement of the user; andresponsive to a determination that the first predefined gesture isevaluated as having a maximum likelihood, matching the detected movementof the user to the first predefined gesture.

In various embodiments, a threshold distance from the real-world objectmay be about an average person's arms length, about 1 ft, about 1 m,and/or the like. In various embodiments, threshold values for likelihoodbased on classification algorithms may be, for example, 95% or higher,90% or higher, 85% or higher, 80% or higher, 75% or higher, and/or thelike. In some cases, threshold values for likelihood based onclassification algorithms may be a more likely than not point, such asabout 51%, any value greater than 50%, etc.

In some instances, rather than a specific predetermined thresholddistance factor, consideration of which real-world object a user isinteracting with may utilize a likelihood factor based on a distancebetween the user and the real-world object. For example, such alikelihood may be higher within a certain range of the user (e.g.,distances that are easily reachable by the user), and may decreaseoutside of this range as the distance increases. The likelihood outputby a classifier may, for example, be weighted by such adistance-to-object likelihood factor to determine an improved likelihoodof a given gesture. In some cases, a threshold may still be applied forthe distance at which the distance-based likelihood factor is near zero,or below a certain minimum level.

In various instances, the first haptic feedback component incommunication with the mobile device may comprise i) a haptic feedbackmodule of the mobile device, or ii) a haptic feedback component of ahaptic device of the user. In various instances, the retrievedinformation identifying the first haptic feedback response may indicatethat the first haptic feedback response is a vibration, and executingthe first haptic feedback response may comprise: controlling the firsthaptic feedback component to generate the vibration of first hapticfeedback response.

In various instances, the retrieved information identifying the firsthaptic feedback response may indicate that the first haptic feedbackresponse is a simulated texture, and executing the first haptic feedbackresponse may comprise: detecting continuing movement of the user; andcontrolling the first haptic feedback component to generate the firsthaptic feedback response based on the detected continuing movement ofthe user. For example, after initial determination that the user istouching a brick wall with their hand, the haptic feedback component maygenerate the haptic response (such as a vibration) as the user continuesto move their hand on the brick wall, and stop when the user removestheir hand from the wall.

In various instances, the retrieved information identifying the firsthaptic feedback response may indicate that the first haptic feedbackresponse is a simulated texture, and executing the first haptic feedbackresponse may comprise: detecting vibrations resulting from continuingcontact between the user and the first real-world object; controllingthe first haptic feedback component to adjust the first haptic feedbackresponse in view of the detected vibrations. As such, vibrationsresulting from the user's interaction with a real world object may befactored into the execution of the haptic feedback response. Forexample, while the user's AR system overlays a first virtual texture(such as animal fur) on the real-world object (such as a brick wall),the haptic feedback component may monitor the vibrations resulting fromthe user touching the brick wall to adjust the haptic feedback responseto improve the simulation of the user “touching” the virtual animal fur(e.g., increasing or decreasing the haptic feedback as appropriate).

In one exemplary scenario, the mobile device may comprise an augmentedreality device, the first predefined gesture may be further associatedwith an augmented reality object presented to the user, the firstprogram action may be an augmented reality user interaction with theaugmented reality object, and the first haptic feedback response maysimulate physical interaction of the user with the augmented realityobject.

Location Based Embodiments

One embodiment of main components of a system for a location-basedapproach is illustrated in FIG. 1. In an embodiment, a system 1002 maycomprise a location sensor 1005, an object database 1010, a gesturesensor 1015, and a gesture classifier 1020.

Location Sensor. The location sensor 1005 may detect the location of theuser in space. The detected location of the user in space may be eitherrelative or absolute. In one embodiment, it may comprise a GPS sensor.In other cases, other sensors may also be used, such as a digitalcompass or an indoor positioning system. Location sensor 1005 may bepart of a smart watch, a smart phone, AR glasses, or any other device inthe system.

Object Database. The object database 1010 may list the locations andproperties of objects of interest within the user's environment. Foreach object of interest in the environment, for example, the databasemay list their location, type, allowed gestures, and resulting actions.In various embodiments, the object database 1010 may reside in thecloud, be local to the system, or comprise a combination of both (e.g.,cached).

Gesture Sensor. The gesture sensor 1015 may detect gestures performed bythe user. It may comprise an accelerometer embedded in a wearable deviceclose to the hand, such as a smart watch or fitness tracker, or thelike. Other options may include a depth camera (e.g., Kinect or leapmotion), an electromyographic sensor (e.g., Myo armband), or any othersystem capable of detecting motion of the hands. These sensors maygenerally be in communication with the mobile device, and may be acomponent of the mobile device or another device associated with theuser.

Gesture Classifier. The gesture classifier 1020 may comprise a softwaresystem that processes the information generated by the gesture sensor1015 and determines which gesture has been performed. This may be doneusing machine learning techniques to learn from observations ofdifferent gesture and non-gesture sensor measurements. A gestureclassifier may also be configured by specifying a set of fixed rules,such as conditions to be met for a certain gesture to be detected. Thissoftware may be executed on any computing device in the system, or insome embodiments in the cloud.

In some embodiments, rules for gesture detection may utilize a weightedsum of the likelihoods, which may give more weight to more reliableclassifiers. The weights may be set dynamically, e.g., by having one ormore classifiers report their confidence in current conditions. In otherembodiments, a threshold for gesture detection by a classifier may bedynamic or otherwise variable. For example, the threshold may be sethigher if a particular gesture may result in a serious (perhapsirreversible) action, such as releasing a “virtual creature” orexecuting a financial transaction.

A flow chart for one embodiment of a location-based approach isillustrated in FIG. 2. The user's location may be detected 205, usingthe location sensor (e.g., GPS) to detect the location of the user. Thesystem may query the object database based on the user's location 210.The database may return the location and type of nearby objects, as wellas the gestures and actions associated with them. Objects may includelandmarks, walls, statues, benches, tables, lamp posts, windows, signs,or any other fixed object that can be touched. In some cases, objectsmay include objects which are not fixed, but whose location can betracked. Gestures may include tapping, brushing, resting, pushing,scratching, poking, displacing a moving part, or any other gesture thatcan be performed against an object. The system may evaluate whether anobject of interest is nearby 215. For example, the system may determinewhether an object of interest is within reach of the user, in someembodiments taking into account the resolution of the location sensor.The system may also take into account the physical characteristics ofthe user, for example if they are known or can be estimated, in order tobetter estimate his or her reach. The reach of a user may, for example,be estimated based on their height and the length of their arms, whichmay in turn be estimated based on their age and gender. If no object ofinterest is nearby, the system may (220) continue updating the userlocation (205) and querying the database (210) until one is found.

Once an object of interest is determined to be nearby, the system may insome embodiments enable gesture detection 225. In such embodiments, thegesture detection system may be disabled until an object of interest hasbeen detected nearby. In other embodiments, the system may keep gesturerecognition enabled at all times. The system may then attempt toclassify a detected gesture 230. For example, the system may run aclassification algorithm to determine which of the gestures permittedagainst the object of interest has been performed. In one embodiment,the classification algorithm may be the result of a machine learningprocess: e.g., sensor data is recorded as a wide range of people performdifferent gestures; this data is used to train a machine learningalgorithm to distinguish between the different gestures or no gesture atall; the algorithm is then capable of indicating the likelihood that agesture has been performed based on sensor data. Some examples of suchtechniques are discussed in Watanabe, et al., “A recognition method forcontinuous gestures with an accelerometer.”, in Proceedings of the 2016ACM International Joint Conference on Pervasive and UbiquitousComputing: Adjunct (UbiComp '16), and Gao, et al., “Deep learning fortactile understanding from visual and haptic data”, 2016 IEEEInternational Conference on Robotics and Automation, 2016, pp. 536-543.In some cases, it may be sufficient to distinguish between the handbeing at rest (motion below a certain threshold of frequency andamplitude) and the hand being active.

The system may then evaluate whether a retrieved predefined gestureagainst the object of interest has been performed 235. For example, thesystem may determine if a gesture has been detected, and if this gestureis associated with the object of interest. If not, the system maycontinue looking for a gesture of interest (240) and updating thelocation of the user periodically.

Once a retrieved predefined gesture for the object of interest isdetected as being performed by the user, the system may perform anaction 240. In an embodiment, the database information specifies theaction that results from the gesture. In the case of an AR game, forexample, the action could be to transfer a digital asset (e.g., amonster, coins, points) to the account of the user and displayingfeedback onscreen. If two objects of interest are detected in proximity,the gesture detector may disambiguate which is interacted with, andwhich action should be taken. It may for example assume that the objectinteracted with is the closest object or the object for which thegesture classifier indicates the greatest likelihood. The system mayalso ignore ambiguous gestures when more than one object is nearby, orwhenever the distance to multiple objects or the likelihood of a gesturehaving been performed against them is too close to disambiguate them.

In some embodiments, the system may optionally produce haptic feedbackin response to the gesture 245. This feedback may be produced on anyhaptic device in contact with the body of the user, including on awearable band or on AR glasses. Any haptic technology may be used,including vibration, deformation, electrical stimulation, stroking,squeezing, variations in temperature, a change in the perceivedcoefficient of friction, and/or the like. The haptic feedback mayindicate that a gesture is being detected (e.g., a continuousvibration), or that a gesture has been detected and has trigged anaction (e.g., a brief, sharp vibration). The haptic feedback may also betied to virtual content that is part of the AR simulation. A vibration,for example, may attempt to simulate the feel of a virtual animal's furwhile touching a real wall. The system may also take into account thenatural haptics produced by the real object as it renders the feel ofthe virtual object. The vibrations produced by a real wall, for example,may possibly be subtracted from the intended vibrations simulating avirtual animal's fur in order to improve the realism of the interaction.

A haptic device may be configured to output a haptic effect comprising avibration, a change in a perceived coefficient of friction, a simulatedtexture, a change in temperature, a stroking sensation, anelectro-tactile effect, or a surface deformation (e.g., a deformation ofa surface associated with the system). Further, some haptic effects mayuse multiple haptic devices of the same or different types in sequenceand/or in concert.

In some embodiments, the haptic device is configured to output a hapticeffect comprising a vibration. The haptic device may comprise, forexample, one or more of a piezoelectric actuator, an electric motor, anelectro-magnetic actuator, a voice coil, a shape memory alloy, anelectro-active polymer, a solenoid, an eccentric rotating mass motor(ERM), or a linear resonant actuator (LRA).

In some embodiments, the haptic device is configured to output a hapticeffect modulating the perceived coefficient of friction of a surfaceassociated with the system. In one embodiment, the haptic devicecomprises an ultrasonic actuator. An ultrasonic actuator may vibrate atan ultrasonic frequency, for example 20 kHz, increasing or reducing theperceived coefficient of an associated surface. In some embodiments, theultrasonic actuator may comprise a piezo-electric material.

In some embodiments, the haptic device uses electrostatic attraction,for example by use of an electrostatic actuator, to output a hapticeffect. The haptic effect may comprise a simulated texture, a simulatedvibration, a stroking sensation, or a perceived change in a coefficientof friction on a surface associated with the system. In someembodiments, the electrostatic actuator may comprise a conducting layerand an insulating layer. The conducting layer may be any semiconductoror other conductive material, such as copper, aluminum, gold, or silver.The insulating layer may be glass, plastic, polymer, or any otherinsulating material. Furthermore, the system may operate theelectrostatic actuator by applying an electric signal, for example an ACsignal, to the conducting layer. In some embodiments, a high-voltageamplifier may generate the AC signal. The electric signal may generate acapacitive coupling between the conducting layer and an object (e.g., auser's finger, head, foot, arm, shoulder, leg, or other body part, or astylus) near or touching the haptic device. Varying the levels ofattraction between the object and the conducting layer can vary thehaptic effect perceived by a user.

In some embodiments, the haptic device comprises a deformation deviceconfigured to output a deformation haptic effect. The deformation hapticeffect may comprise raising or lowering portions of a surface associatedwith the system. For example, the deformation haptic effect may compriseraising portions of a surface of an object to generate a bumpy texture.In some embodiments, the deformation haptic effect may comprise bending,folding, rolling, twisting, squeezing, flexing, changing the shape of,or otherwise deforming a surface associated with the system. Forexample, the deformation haptic effect may apply a force on the systemor a surface associated with the system, causing it to bend, fold, roll,twist, squeeze, flex, change shape, or otherwise deform.

In some embodiments, the haptic device comprises fluid configured foroutputting a deformation haptic effect (e.g., for bending or deformingthe system or a surface associated with the system). For example, thefluid may comprise a smart gel. A smart gel comprises a fluid withmechanical or structural properties that change in response to astimulus or stimuli (e.g., an electric field, a magnetic field,temperature, ultraviolet light, shaking, or a pH variation). Forinstance, in response to a stimulus, a smart gel may change instiffness, volume, transparency, and/or color. In some embodiments,stiffness may comprise the resistance of a surface associated with thesystem against deformation. In some embodiments, one or more wires maybe embedded in or coupled to the smart gel. As current runs through thewires, heat is emitted, causing the smart gel to expand or contract.This may cause the system or a surface associated with the system todeform.

As another example, the fluid may comprise a rheological (e.g., amagneto-rheological or electro-rheological) fluid. A rheological fluidcomprises metal particles (e.g., iron particles) suspended in a fluid(e.g., oil or water). In response to an electric or magnetic field, theorder of the molecules in the fluid may realign, changing the overalldamping and/or viscosity of the fluid. This may cause the system or asurface associated with the system to deform.

In some embodiments, the haptic device comprises a mechanicaldeformation device. For example, in some embodiments, the haptic devicemay comprise an actuator coupled to an arm that rotates a deformationcomponent. The deformation component may comprise, for example, an oval,starburst, or corrugated shape. The deformation component may beconfigured to move a surface associated with the system at some rotationangles but not others. The actuator may comprise a piezo-electricactuator, rotating/linear actuator, solenoid, an electroactive polymeractuator, macro fiber composite (MFC) actuator, shape memory alloy (SMA)actuator, and/or other actuator.

Further, other techniques or methods can be used to deform a surfaceassociated with the system. For example, the haptic device may comprisea flexible surface layer configured to deform its surface or vary itstexture based upon contact from a surface reconfigurable hapticsubstrate (including, but not limited to, e.g., fibers, nanotubes,electroactive polymers, piezoelectric elements, or shape memory alloys).In some embodiments, the haptic device is deformed, for example, with adeforming mechanism (e.g., a motor coupled to wires), air or fluidpockets, local deformation of materials, resonant mechanical elements,piezoelectric materials, micro-electromechanical systems (“MEMS”)elements or pumps, thermal fluid pockets, variable porosity membranes,or laminar flow modulation.

In some embodiments, the haptic device is configured to remotely projecthaptic effects to a user. For example, the haptic device may compriseone or more jets configured to emit materials (e.g., solids, liquids,gasses, or plasmas) toward the user (e.g., toward the back of the user'shand). In one such embodiment, the haptic device comprises a gas jetconfigured to emit puffs or streams of oxygen, nitrogen, carbon dioxide,or carbon monoxide with varying characteristics upon receipt of thehaptic signal. As another example, the haptic device may comprise one ormore ultrasonic transducers or speakers configured to project pressurewaves in the direction of the user. In one such embodiment, upon theuser interacting with an object, the system may cause the haptic deviceto emit a concentrated pressure wave toward the user. The concentratedpressure wave may vibrate a portion of the user's body (e.g., the user'shand).

Sensor Based Embodiments

One embodiment of main components of a system for a sensor-basedapproach is illustrated in FIG. 3. The system 302 may comprise avibration sensor 305 and a vibration classifier 310.

Vibration Sensor 305 may detect interactions between the hand and anobject through vibrations. In an embodiment, the vibration sensorcomprises an accelerometer, such as embedded in a smart watch, a smartring, or fitness tracker. As the user interacts with objects havingdifferent physical properties, different vibration patterns are producedon the accelerometer (or other vibration sensor). In some embodiments,the vibration sensor may comprise an alternative sensor, such as amicrophone (for example, a microphone in a smart ring may detectparticular sounds as a user's finger brushes against a particulartextured surface—like a brick wall as opposed to a wood wall).

Vibration Classifier 310 may comprise a software component whichanalyses the vibrations produced and estimates the material propertiesof the object touched and the gesture performed against it.

In some embodiments, rules for vibration classification may utilize aweighted sum of the likelihoods, which may give more weight to morereliable classifiers. The weights may be set dynamically, e.g., byhaving one or more classifiers report their confidence in currentconditions. In other embodiments, a threshold for vibrationdetermination by a classifier may be dynamic or otherwise variable. Forexample, the threshold may be set higher if a particular vibrationpattern (either independently or in association with a particulardetected user motion, such as a hand stroke against a brick wall asopposed to a hand stroke on a wood wall) is associated with a serious(perhaps irreversible) action, such as releasing a “virtual creature” orexecuting a financial transaction.

FIG. 4 illustrates a flow chart of one embodiment of a sensor-basedapproach. In an embodiment, the system may specify target interactions405. For example, the AR application may specify the target interactionsgiven a current state of the software. Target interactions may includecombinations of gesture and material properties that are likely to bedetectable, such as tapping against a hard surface, brushing against arough surface, poking a soft object, or the like. The system may thenmonitor vibrations 410. Vibration data is collected by the vibrationsensor. The system may then detect target interactions 415. Thevibration classifier may be used to detect whether any of the targetinteractions has been detected. This may be similar to theclassification of gestures, as described above. The classificationalgorithm may be trained using machine learning techniques to detectvarious vibrations patterns that result from specific interactions withspecific objects. The system may, for example and without limitation, betrained to detect brushing against a brick wall. The classificationalgorithms used in some embodiments may be similar to those discussed inRomano and Kuchenbecker and Gao.

The system may then determine whether a target interaction has beendetected 420. If not, the system may keep monitoring vibrations until atarget interaction is detected 425. If a target interaction is detected,the system may perform an action 430. In an embodiment, the system mayhave information which specifies the action that results from thedetected interaction. In some embodiments, step 430 may be similar tostep 240 from FIG. 2.

In some embodiments, the system may optionally produce haptic feedback435. In some cases, this feedback may be similar to step 245 from FIG.2. Haptic feedback may be produced on any haptic device in contact withthe body of the user, including on a wearable band or on AR glasses. Anyhaptic technology may be used, including vibration, deformation,electrical stimulation, stroking, squeezing, variations in temperature,a change in the perceived coefficient of friction, and/or the like. Thehaptic feedback may indicate that a target interaction is being detected(e.g., a continuous vibration), or that a target interaction has beendetected and has trigged an action (e.g., a brief, sharp vibration). Thehaptic feedback may also be tied to virtual content that is part of anAR simulation.

In some embodiments, the location based and sensor based approaches maybe combined. In some cases, this may result in improved outcomes for thesystem. In an embodiment, the “target interaction” of the sensor-basedapproach may be specified by querying the object database of thelocation-based approach for nearby interactions of interest. Theclassification resulting from each approach may similarly be fused toproduce more accurate results. A combined likelihood of a gesture may,for example, be obtained by taking a maximum, average, or a minimum ofthe likelihoods produced by the location-based and sensor-basedapproaches (or a weighted sum combination, wherein the weight of a givenclassifier may be predetermined or dynamic, such as based on aconfidence of the classifier result). Alternatively, the sensor-basedapproach may be executed only when a gesture has been detected by thelocation-based approach in order to confirm the classification, therebyreducing computational requirements.

In an exemplary embodiment, as illustrated in FIGS. 5A-5C, a user may bewearing an AR device and interacting with an environment. As shown inFIG. 5A, within a user's real-world field of view 501 may be a brickwall 525. The user may also see their own hand (505), and be wearing asmartwatch 510. The smartwatch 510 may include a vibration sensorcomponent and a haptic feedback component. The smartwatch 510 may be incommunication with the user's AR device. In an AR view 503 presented tothe user, as shown in FIG. 5B, the brick wall 525 may be overlaid withan AR object, such as a virtual textured surface 530. The virtualtextured surface 530 may, for example, be covered in virtual “fur.” Aspreviously discussed, the AR device (or another device or sensor incommunication with the AR device) may determine the user's location (orgeolocation), and the AR device may retrieve information identifying thebrick wall 525, a stroking motion of the user's hand associated with thebrick wall 525, an AR program action associated with the stroking motionof the user's hand, and a haptic feedback response to simulate touchingfur. In one instance, the AR environment presented to the user may bepart of an AR game, and the user may stroke their hand against ARobjects to collect materials within the game. In this case, the programaction associated with the user stroking their hand against virtual fur(530) may be to collect the virtual fur (which may be later used in theAR game).

As shown in FIG. 5C, the user may move (535) their hand against thereal-world brick wall 525 with the overlaid virtual fur texture 530 inthe stroking motion associated with the brick wall. This movement (535)may be detected by one or more sensors in communication with the ARdevice. The detected movement may then be matched to the predefinedstroking motion of the user's hand. If there is more than one retrievedpredefined gesture, the system may determine which gesture the detectedmovement matches, if any. The detected movement may be evaluated in someinstances by a classification algorithm to determine a likelihood thatthe detected movement matches a given predefined gesture. Here, as thedetected movement is matched to the predefined stroking motion of theuser's hand, the program action and haptic feedback response associatedwith the stroking motion may be selected and executed. As such, the ARdevice may collect the virtual fur as the user moves their hand againstthe real-world brick wall with the virtual fur texture AR overlay, andthe haptic feedback response (540) to simulate touching fur may begenerated and controlled by the haptic feedback component of thesmartwatch.

Various instances may further include retrieving locations (orgeolocations) of real-world objects, textures associated with real-worldobjects, vibration profiles of predefined gestures (such as associatedwith a particular motion, a particular motion on a particular texture,etc.), and/or the like.

In some further instances, a vibration sensor, such as a component ofthe smartwatch 510, may detect vibrations caused by the user's tactileinteractions with real-world objects such as the brick wall 525. In somecases, these vibrations may be used in identification of a gesture beingperformed by the user (e.g., is the user stroking their hand on a brickwall or a wood-paneled wall). Detected vibrations may also be used inidentification of a particular real-world object being interacted withby the user (e.g., if the user is within a threshold distance of areal-world object, detected vibrations may be used to determine that theuser is touching an object, so the user must stroke the real-world wall,not just perform the motion in the air).

In various instances, such as where the haptic feedback responsesimulates a virtual texture, detected vibrations from the user'sinteraction with the real-world object may be factored into the controlof the haptic feedback component generating the haptic feedbackresponse. For example, the vibrations generated by the user strokingtheir hand on the brick wall may increase or decrease an intensity orstrength of a particular haptic feedback response as a virtual furtexture is simulated.

In an alternative scenario, rather than a brick-wall, the real-worldobject may be a smooth plaster wall. And rather than a simulated virtualfur texture, the simulated texture may be a simulated virtual brick walltexture. In such a scenario, the haptic feedback response 540 maycontrol the haptic feedback component to simulate the feel of a brickwall (the virtual texture), such as with controlled vibrations or otherhaptic feedback.

FIG. 6 illustrates a flow diagram for an embodiment such as that inFIGS. 5A-5C. At some point, the user location may be detected (605), andbased on the determined location of the user the user's mobile devicemay retrieve from a database or remote server information (610)identifying gesture combinations, which may each comprise (i) at least afirst nearby real-world object, (ii) at least a first predefined gestureassociated with the first real-world object, and (iii) at least a firstprogram action associated with the first predefined gesture. In someembodiments, the mobile device may also retrieve at least a first hapticfeedback response associated with the first program action. Gesturedetection may operate to track the movements of the user, and at somepoint a gesture motion of the user may be detected (615). The detectedgesture motion may in some embodiments also include detected vibrationsof the user's tactile interaction with a real-world object. Based on thedetected gesture motion, classification of the detected gesture may beattempted (620), such as with a classification algorithm as previouslydiscussed. If the classified gesture matches a retrieved predefinedgesture (625), the associated program action associated with the matchedgesture may be performed (635). In some embodiments, where a hapticfeedback response associated with the program action is also retrieved,the retrieved haptic feedback response may also be produced, generated,or otherwise controlled (640). If the detected gesture does not match aretrieved gesture, then gesture detection and classification processesmay be repeated. In some embodiments, the location of the real-worldobject may also be retrieved, and utilized in evaluating or classifyingdetected gesture motions.

Further Embodiments

In some embodiments, systems and method disclosed use an accelerometerplaced near the hand, such as in a smartwatch or activity tracker.However, in other embodiments, there may be several other possibilitiesfor gesture detection. For example, gesture sensing technologies mayinclude computer vision used to detect motion of the hand, with orwithout fiducial markers. The motion of the hands may also be trackedusing electromagnetic markers. Gestures may also be inferred from muscleactivation. A graspable device, such as a wand-like game controller, mayalso be used to detect gestures.

In some embodiments, the sensor-based approaches discussed above rely onan accelerometer to sense the properties of the material being touched.In other embodiments, it may, however, also be possible to identify thematerial using other sensors such as a wearable camera or a radar-basedsensor (for example as discussed in Yeo, et al., “RadarCat: RadarCategorization for Input & Interaction.”, Proceedings of the 29th AnnualSymposium on User Interface Software and Technology, 833-841).

In some embodiments, data about tactile gestures being made and thematerial properties of objects being touched may be geo-localized andcollected at a cloud server for utilization. This data may be used tobuild and continuously improve a database of expected gestures andmaterial properties at different locations. The database may, forexample, be initialized with a generic model for the vibration patternexpected from brushing against a wall. As brushing gestures are detectedagainst a specific wall, the vibration patterns recorded can be uploadedto the database and used to re-train the model for this specific wall.Over time, the database may learn and improve its recognition ofgestures against this specific wall. In some cases, the threshold fordetection may initially be set low, and gradually increase as detectionaccuracy is improved with additional data.

While systems and methods set forth herein are described primarily inrelation to visual AR embodiments, such as AR glasses and smartphones,in some other embodiments the systems and methods may also oralternatively be used without any visual feedback. An AR application mayinstead provide cues and feedback with either audio or haptic feedback,or both. A player, for example, may feel different surfaces in hisenvironment until a specific haptic sensation is produced on hissmartwatch, indicating that a virtual item has been picked up and pointscollected. Similarly, the player may be prompted to explore hisenvironment using spoken audio (e.g., “look for a brick wall!”) andsound effects (e.g., a ‘pop’ sound when an item is picked up). Thesource of the sound may be on the user's body (e.g., headphones,wearable speaker) or in the environment (e.g., a stationary speaker).

In some embodiments, systems and methods set forth herein may beextended to facilitate the alignment of virtual content to real objects.Detected tactile gestures may contain information that can be used forregistration. More specifically, a real environment may limit themovements of a user in ways that can be captured within the methods setforth herein. A brushing motion against a wall, for example, may revealthe orientation of the wall relative to the user and allow virtualcontent to be placed on it. Similarly, tapping against the wall mayreveal the location of at least one point of contact. In some cases, thevisual motion in a video feed may be correlated with a registeredinteraction.

In some embodiments, the systems and methods set forth herein may beapplied to VR use cases. For example, a sensor-based approach may beused to detect interactions with objects in the real world while a useris immersed in a VR environment. These interactions may trigger actionsin the VR environment, or be mapped to interactions with matchingvirtual objects. When detecting that the user is touching a wall, forexample, the VR simulation may display a virtual wall that is onlyvisible when touched. Touching a keyboard, on the other hand, may pop-upa typing window. A location-based approach may be incorporated in acontext where a VR environment matches physical objects in the realworld, such as by mapping every wall, door, or other object in the realenvironment to virtual objects in the VR environment. Tactile gesturesmay then be detected as described above, and mapped to actions in thevirtual environment. The augmentation of tactile interactions withartificial haptic feedback may also be utilized in VR use cases. Realworld objects may, for example, be used as props for VR objects andaugmented with tactile feedback that improves realism. Vibrationssimulating a rough surface may, for example, be produced when touching asmooth table that is shown to be rough in a VR environment.

In one exemplary scenario, a player may open an AR game app on theirsmartphone. A tip may pop-up on the screen: “Monsters hide in trees . .. Can you find one?” The user may start walking and look for a tree.Once a tree is found, the user may stop in front of it and brush theirhand against the tree trunk. They may feel a vibration of increasingintensity on their smartwatch, and then a “pop” once a monster has beenfound. In a location-based approach, the app may monitor a GPS signal ofthe smartphone and query an online database for nearby trees. When atree is detected within reach of the player (given the resolution of theGPS signal), the app begins monitoring the accelerometer signal from theplayer's smartwatch. The app looks for a motion pattern that matchesbrushing against a tree trunk. If the pattern is detected, a counterbegins and a command is sent to the watch to output a vibration patternindicating progress. Once the counter has reached a preset level (e.g.,5 seconds), another command is sent to the watch to produce a vibratingpop and the app releases the virtual creature on the screen. In asensor-based approach, the app may monitor the signal from anaccelerometer inside the user's smartwatch. The signal may be processedby an algorithm to determine the likelihood that a tree trunk is beingtouched. Vibration patterns may be produced on the watch when a matchingpattern is first detected, and also once the pattern has been ongoingfor a preset amount of time. In a combined approach, the app may monitora GPS signal of the smartphone and query an online database for nearbytrees. When a tree is detected within reach of the player (given theresolution of the GPS signal), the app begins monitoring theaccelerometer signal from the player's smartwatch in order to detectbrushing against the tree. The algorithms from the location-basedapproach are used to determine the likelihood that the motion patternmatches brushing against the tree's trunk, as described above. Thealgorithms from the sensor-based approach are used to determine thelikelihood that the vibration pattern matches brushing against thetree's trunk, as described above. Both likelihoods are combined todetermine a single likelihood, such as by taking the average or minimumof the two likelihoods obtained with the location-based and sensor-basedapproaches. Vibration patterns are once again produced on the watch whena matching pattern is first detected, and once it has been ongoing for apreset amount of time.

In some additional scenarios, once the app detects that the player istouching the bark of a tree, it may decide whether or not a monster ishiding in this tree, for example either randomly or using a database. Ifa monster is not present, no action will be taken. If a monster ispresent, a vibration pattern may be produced on a watch whenever theplayer's hand crosses roughly the center of the gesture area, giving theillusion of a localized object. In one case, after three repetitions,the monster may be released in the app.

The release of a monster may be triggered by a wide range of tactilegestures against objects in the environment, such as: tapping orbrushing against a tree's trunk; running a hand through a tree or bush'sbranches; running a hand through grass, sand, or water; tapping orbrushing against a statue in a park; knocking on a wall or door;brushing against a surface with a distinctive texture such as a brickwall, a wooden bench, a tree or a plant.

In another exemplary use case, a user may open an AR Tourism app ontheir smartwatch and walk into a park. As they slide their hand around astatue at the center of the park, they may feel a vibration on theirwatch and look at it to see a description of the statue and itshistorical significance. In a location-based approach, the app monitorsthe GPS signal of the smartphone and queries an online database fornearby points of interest. Whenever a point of interest is within reach,the app queries the database to retrieve predefined gestures which maybe performed against the point of interest. It then may monitor anaccelerometer signal of the smartwatch to determine whether one of theretrieved gestures is being performed. When a retrieved gesture isdetected, further retrieved elements (such as a program action and anoptional haptic feedback response each associated with the predefinedgesture and/or real-world object) may be executed.

In another exemplary use case, systems and methods set forth herein maybe utilized in an internet of things (IoT) transaction. For example, auser may open a music playing app on their smartphone. While holding avirtual button in the UI, they may touch a nearby speaker to start musicplaying from that speaker. In a combined approach, the music app may usethe GPS signal of the phone to determine what speakers are nearby andtheir physical properties. It may then use an accelerometer in thesmartwatch to monitor vibrations resulting from the user's contact withan object. When a contact is detected, the signal may be processed todetermine material properties of the contacted object. If these match anearby speaker, that speaker may be selected to play the music.

Still further use cases may utilize the systems and methods set forthherein. For example, certain objects in an environment may deliverinformation haptically when touched. For example, when it is detectedthat a user has grabbed a pole for a bus sign (such as by locationevaluation of the user and the pole, and vibration detection to confirmthe user has touched the pole), a time until the next bus may bedisplayed on the user's smartwatch, and the smartwatch may vibrate todraw the user's attention to the displayed wait time. Similarly, toassist vision or hearing impaired users, detection that the user hastouched or grabbed a pole or post at an intersection with a crosswalkmay retrieve (if available to the device) a current status of acrosswalk indicator and provide a haptic feedback response to the userfor that current status (e.g., a sequence of vibrations if it is nottime to cross, or a continuous vibration when the walk signal is activethat may change in intensity as the time of the walk signal countsdown).

Objects in an environment may also be used as waypoints in a game,during exercise, or the like. A runner, for example, may touch a tree orother specific object in the environment as a target marker and feelfeedback about their performance. For example, a tap gesture against asolid real-world object may be associated with a program action torecord a lap time for the runner and compare that lap time to a pacepreset by the runner, and a haptic feedback response may deliveralternative vibration sequences for whether the runner met or beat thepace time or was slower than the pace time.

Making a gesture against an object may also trigger an action, either inthe physical or digital world. Haptic feedback can provide guidance(e.g., indicate that the gesture is possible, that it is being detected,or what action will result) or confirm that the action has beentriggered. Examples of actions include, but are not limited to: openingor unlocking a door or container; controlling the intensity of alight, asound, a water flow, a heating element, etc.; using objects as a remotecontrol; information queries; and/or the like.

For example, opening or unlocking a door or container may havepredefined triggering gestures including but not limited to: sliding ahand or foot upwards against a garage door to open it; sliding a handhorizontally against a sliding door to open it; making a rotatinggesture against a lock to unlock it; sliding a hand down against a carwindow to open it; etc. In one case, when it is detected that the userhas moved their hand upwards against a garage door, a retrievedpredefined gesture of the upwards hand movement associated with thegarage door may have an associated program action to open the garagedoor. Alternatively, a downward hand motion against a wall next to thegarage door may be associated with a program action to close the garagedoor, and may be similarly executed based on detection of the downwardhand motion.

For example, controlling the intensity of alight, a sound, a water flow,a heating element, or the like may have predefined triggering gesturesincluding but not limited to: sliding a hand against a light post toadjust the light intensity; sliding a hand against a wall next to abathtub to control water flow; sliding a hand up and down a wall tocontrol the lighting in a room; controlling the volume of outdoorspeakers in a backyard by sliding a hand up and down a tree; moving ahand in the water next to a pool's waterjet to activate them; etc.

For example, using a nearby object as a remote control may havepredefined triggering gestures including but not limited to: sliding ahand against a sofa to change the channel or control the sound on atelevision; extending a game console controller to other objects in theroom, e.g., tapping against the coffee table or sofa; gesturing againststreet furniture (e.g., a mailbox) to interact with a large screendisplay (e.g., Times Square); allowing a crowd of people to play musictogether by interacting with street furniture and other objects in apublic space, e.g., control percussion by tapping against a bench or thepitch of a sound by touching a statue; etc.

Other examples of actions and their predefined trigger gestures includebut are not limited to: gesturing against a table or menu in arestaurant to call the waiter; gently tapping twice on a shared car(e.g., car2go) to start using it; making a specific gesture againststreet furniture (e.g., a lamp post) to order a private car service(e.g., Lyft or Uber) at that location; paying for a parking meter bytapping twice against your car or against the park meter; interactingwith an advertisement (e.g., a video display in a bus shelter) bytouching nearby objects (e.g., touching the walls of the bus shelter);etc.

To perform an information query, an object may be touched or interactedas the predefined gesture to trigger the display of information. Invarious cases, the display can be visual (e.g., a pop-up display in AR),audio (e.g., an audio recording or computer-generated voice), or haptic(e.g., a vibration or sequences of vibrations that communicate aparticular meaning). In various scenarios, an information action and itspredefined trigger gesture may include but are not limited to: touchinga door, window or wall to indicate if someone is inside, if the personinside is busy, if a store is open, etc.; touching an umbrella, an itemof clothing, or a door indicates the weather, e.g., whether umbrella ora warm jacket is needed; touching a post or traffic light at anintersection indicates whether it is safe to cross, e.g., for visuallyimpaired pedestrians; touching a bus shelter or a bus stop signindicates when the next bus is coming; touching a product in a storeindicates whether it should be purchased based on price comparisons,home inventory, or other criteria; touching a plant indicates whether itshould be watered, based on IoT sensors or the like; touching a pictureframe triggers a recorded message or opens a chat window with the personin the picture; touching an advertisement (e.g., a poster) opens up avideo or webpage; touching the water in a pool indicates the water'stemperature, its chlorine level, etc.; and/or the like.

In some cases, gestures against objects can be used to establish linksor transactions between multiple devices and/or objects. Haptic feedbackcan confirm that the objects have been selected and that a link ortransaction has been established or concluded. A distinct vibration can,for example, be produced to indicate each event. Examples include butare not limited to the following. Touching an audio-video source and aplayback device to link them: for example, display a video stored on acamera on a television by first touching the camera and then thetelevision; play music from a smartphone on wireless speaker by touchingthe smartphone with one hand and the speaker with the other; etc. Makinga payment by touching the source of the money or a proxy for it (e.g., acredit card, a wallet) and then touching the payment system (e.g., avending machine, a card reader).

In addition to previously described examples, in some cases systems andmethods disclosed herein may be used in interactive tactile games whichinvolve having people hide virtual objects in the environment and havingothers look for them by touch (for example), with hints. For example, aplayer could hide a virtual object in a tree and tell another playerthat it is hidden in something rough and brown. A game played in thereal world may similarly provide clues about a player's whereabouts withhaptic feedback that can be felt by touching objects. When playinghide-and-seek, for example, a player may feel whether another player hasbeen somewhere and how long ago by feeling haptic feedback in theirshoes as they touch the ground. This information may, for example, beindicated using a vibrotactile or thermal haptic effect that diminishesin intensity based on the time since another player has been at alocation. AR games may be controlled by interacting with objects in theenvironment. For example, a boat projected on a water puddle may beplayed with by making gestures in the water with a hand or foot, andhaptic feedback may be felt in a user's smartwatch. In a particularcase, a user may make gestures in the water, and the system may useeither the location or the vibrations that are produced to deduce thatthe gestures are being made in the water, and map these gestures todirection and speed controls of the projected boat. Haptic feedback maybe produced on the user's smartwatch to simulate the motor of the boator the feel of water against its hull. In another use case, a player mayuse a game controller or objects in the room around the player tointeract with a console game. For example, a door can be kicked down inthe game by the player gently kicking against a coffee table or thefloor. Or spells can be cast by making gestures against the surface ofthe sofa where the user is sitting. The system may have a map of theroom and/or physical properties of the nearby objects, and can use theinformation to detect gestures against them. A sensor in a shoe, forexample, can detect kicking the floor or a coffee table. Sensors in awristband can detect brushing against the sofa. These actions are mappedto functions in the game, and haptic feedback can be used to confirm theaction or simulate it more realistically.

AR may also be used to present promotional haptic AR scenarios. Forexample, a retail store chain may run a promotion where as people passby one of its physical stores, they can make a circular gesture againstthe store's window to get a chance to win a rebate coupon. As peoplemake the gesture, they may feel a scratchy texture that gradually goesaway, similar to a real-world scratch-off lottery ticket. In some cases,if they've won, a sensation may be felt similar to a coin against ascratch-off ticket once the scratchy texture has been removed. If peopleare wearing AR glasses, they may also see AR representations of thescratch-off surface and the “coin” as they make the circular gestureagainst the window. Embodiments such as this do not call for astorefront to be instrumented in any way, utilizing informationaccessible to user devices to present the promotional haptic ARscenario. As the user gets close to the store's window, his watchdetects his position and queries a database to know whether a store ofthe retail chain is nearby. In fact, the embodiment could even beapplied to the stores of competitors. When the user makes a gestureagainst the window, an accelerometer in the wristband detects thecircular gesture and an algorithm concludes that the gestures must bemade against the window based on its orientation and proximity. Thedevice then produces vibrations that simulate the feel of a scratchsurface with diminishing intensity. It queries a server of the retailchain that determines whether or not the user has won. If it has, itproduces a haptic effect that resembles a coin. If not, it stopsproducing haptic feedback. In other cases, an alternative promotion mayinvolve a user finding a phone booth and rubbing their hand against itto win free air time from a wireless provider. In another example, apromotion from a music streaming company may involve rubbing a handagainst different objects to hear music related to that object (e.g., asong about a bridge when touching it) and a chance to win a prize.

In another use case, systems and methods set forth herein may be usedfor interactive advertisements. For example, a user may approach aposter for a new racing console game. Based on their location and adatabase storage having a gesture associated with the poster, the user'smobile device may detect that the user is touching the poster (forexample, based on the user's location, the location of the poster, and avibration detection indicating an object has been touched). As theytouch the poster, they may feel a haptic feedback response of avibration on their phone as a webpage for the game is opened up on thephone. As they continue to move their hands on the poster, the user mayalso feel a haptic feedback response of a sequence of haptic effects(such as vibrations) on their smartwatch which may simulate the feel ofthe motor of a racecar. If the user is wearing AR glasses, they may alsosee that the poster is interactive, and a program action may beassociated with the gesture of sliding their hands left or right on theposter or touching against a brick wall on its left or right to make anAR virtual car move on the interactive poster. Again, no additionalinstrumentation is called for beyond a user's own devices. A wearable ormobile device carried by the user detects their location and determinesthat they are close to the poster, and detects their gestures todetermine whether a relevant input has been made against the poster(such as tapping, sliding against it, or sliding against the texturedsurface of the wall next to it). These gestures may trigger actions onthe user's watch, smartphone, AR headset, or other device.

Systems and methods set forth herein may also be utilized in homeautomation, such as with IoT systems. Rather than controlling IoTsystems with a screen based interface of their phone or other device, auser may more quickly control IoT system functionalities by touchingobjects in the environment to which specific functions are tied. Forexample, a garage door can be opened by brushing a foot upwards againstit, which may be convenient when carrying bags. Or lights in a backyardmay be dimmed by sliding a hand up and down against a tree, without theuser walking over to a mechanical switch. Or waterjets in a pool may beturned on or off by waving towards or away from the vents in the water.As in other use cases, the user's devices (and their sensors) are usedto detect interactions based on the user's location and/or the textureof objects. A sensor in the user's shoes, for example, can detectbrushing against the garage door, confirm based on the location of theuser, and trigger opening of the garage door. In various cases, avibration actuator (a particular type of haptic feedback component) in awearable device (e.g., wrist or ankle band) may produce detents toindicate that a mechanism is being activated (e.g., different lightsettings) or a strong detent to indicate that something has been fullyactivated (e.g., opening of a door, especially if slightly delayed).

Systems and methods set forth herein may be used with car share programsto book a vehicle by a user finding an available vehicle and tappingagainst it twice, causing a system to automatically book the vehicle andunlock it for a user. Or with ride services, where a user may make astored contact sequence (such as a simple 2 taps—pause—2 taps sequence)against any kind of street furniture to request a pickup, and the usermay receive haptic feedback if a ride service car is selected to pickthem up.

In an embodiment, there is a method of initiating a program action in anaugmented reality after detecting a tactile interaction comprising:determining a location of a first user with at least one sensor of atleast a first mobile device; retrieving at least a first object entryfrom an object database based on the determined location of the firstuser, the first object entry comprising an identifier of a firstproximate object, at least one gesture action associated with the firstproximate object, and at least one program action associated with eachof the gesture actions; detecting a first gesture performed by the firstuser with at least one sensor of the mobile device; matching thedetected first gesture to at least one gesture action in the firstobject entry; and executing the at least one program action associatedwith the matched at least one gesture action. The method may alsoinclude wherein the first object entry further comprises a hapticfeedback response associated with at least one of the gesture actions.The method may also include executing the haptic feedback responseassociated with the matched at least one gesture action. The method mayalso include wherein the haptic feedback response indicates a gesture isbeing detected. The method may also include wherein the haptic feedbackresponse indicates a gesture has been detected and the at least oneprogram action associated with said gesture has been executed. Themethod may also include wherein the haptic feedback response is tied tovirtual content which is part of an augmented reality environmentgenerated by the mobile device. The method may also include wherein thehaptic feedback response simulates a sensation of a virtual contentobject associated with the first proximate object. The method may alsoinclude dynamically adjusting the haptic feedback response to accountfor natural haptics produced by the first proximate object as the firstuser interacts with the first proximate object. The method may alsoinclude wherein the location of the first user is determined by a GPSsensor of the mobile device. The method may also include wherein theobject database is local to the mobile device, or wherein the objectdatabase resides in the cloud. The method may also include whereindetecting the first gesture comprises detecting the first gesture withan accelerometer of the at least first mobile device. The method mayalso include wherein matching the detected first gesture to at least onegesture action in the first object entry comprises evaluating thedetected sensor data of the first gesture with a classificationalgorithm to determine which of the at least one gesture action has alikelihood greater than a threshold value. The method may also include,responsive to a determination that at least two objects of interest areproximate to the first user and each is associated with a gesture actionmatched to the first detected gesture, executing the at least oneprogram action of only the closest of the at least two objects ofinterest. The method may also include, responsive to a determinationthat at least two objects of interest are proximate to the first userand each is associated with a gesture action matched to the firstdetected gesture, executing the at least one program action of only theone of the at least two objects of interest which has a greatestlikelihood as determined by a gesture classification algorithm. Themethod may also include enabling a gesture detection mode of the mobiledevice only when the first user is proximate to at least one object ofinterest. The method may also include wherein retrieving at least thefirst object entry from the object database based on the determinedlocation of the first user comprises: calculating, for at least oneobject entry in the object database, a distance from said object to thefirst user's determined location; wherein the retrieved at least firstobject entry comprises at least one object entry having a calculateddistance less than a predefined threshold value. The method may alsoinclude wherein the first object entry further comprises at least afirst tactile interaction associated with the first object and at leastone program action associated with the first tactile interaction. Themethod may also include detecting movements and vibrations caused by thefirst user interacting with the first object, with at least one sensorof the mobile device; and responsive to a determination that detectedmovements and vibrations indicate a particular tactile interaction hasoccurred, initiating a first program action associated with said tactileinteraction in a memory of the mobile device. The method may alsoinclude wherein determining that detected movements and vibrationsindicate the particular tactile interaction has occurred comprisesevaluating the detected sensor data of the first user interacting withthe first object with a classification algorithm to determine theparticular tactile interaction of a plurality of tactile interactionswhich has a likelihood greater than a threshold value. The method mayalso include responsive to the determination, initiating a hapticresponse associated with said tactile interaction. The method may alsoinclude wherein a tactile interaction comprises a combination of gestureand material properties. The method may also include wherein aparticular tactile interaction comprises the first user tapping againsta hard surface, wherein a particular tactile interaction comprises thefirst user brushing against a rough surface, and/or wherein a particulartactile interaction comprises the first user poking a soft object. Themethod may also include wherein the at least one sensor of the mobiledevice comprises an accelerometer. The method may also include detectingmovements and vibrations caused by the first user interacting with thefirst object, with at least one sensor of the mobile device. The methodmay also include wherein matching the detected first gesture to at leastone gesture action in the first object entry comprises: evaluating withat least one classification algorithm likelihoods of the at least onegesture action based separately on each of the detected first gestureand the detected movements and vibrations; taking an average of theseparate likelihoods; and selecting the first gesture action if theaverage of the likelihoods exceeds a threshold value. The method mayalso include wherein matching the detected first gesture to at least onegesture action in the first object entry comprises: evaluating with atleast one classification algorithm likelihoods of the at least onegesture action based separately on each of the detected first gestureand the detected movements and vibrations; selecting the minimum valueof the separate likelihoods; and selecting the first gesture action ifthe minimum value of the likelihoods exceeds a threshold value.

In one embodiment, there is a method, comprising: initiating detectionmode of a mobile device to await detecting of a tactile interaction of afirst user with a first object; detecting movements and vibrationscaused by the first user interacting with the first object in a naturalenvironment, with at least one sensor of the mobile device; andresponsive to a determination that detected movements and vibrationsindicate a particular tactile interaction has occurred, initiating afirst program action associated with said tactile interaction in amemory of the mobile device. The method may also include determining alocation of the first user with at least one sensor of the mobiledevice; and retrieving at least a first object entry associated with thefirst object from an object database based on the determined location ofthe first user; wherein the first object entry comprises at least afirst tactile interaction associated with the first object, and at leastthe first program action associated with the first tactile interaction.The method may also include wherein the detection mode of the mobiledevice is initiated responsive to a determination that at least oneobject is proximate to the first user. The method may also includewherein a tactile interaction comprises a combination of gesture andmaterial properties. The method may also include wherein a particulartactile interaction comprises the first user tapping against a hardsurface, wherein a particular tactile interaction comprises the firstuser brushing against a rough surface, and/or wherein a particulartactile interaction comprises the first user poking a soft object. Themethod may also include responsive to the determination, initiating ahaptic response associated with said tactile interaction. The method mayalso include wherein the at least one sensor of the mobile devicecomprises an accelerometer. The method may also include whereindetermining that detected movements and vibrations indicate theparticular tactile interaction has occurred comprises evaluating thedetected sensor data of the first user interacting with the first objectwith a classification algorithm to determine the particular tactileinteraction of a plurality of tactile interactions which has alikelihood greater than a threshold value.

In one embodiment, there is a system comprising a processor and anon-transitory storage medium storing instructions operative, whenexecuted on the processor, to perform functions including: determining alocation of a first user with at least one sensor of a mobile device;retrieving at least a first object entry from an object database basedon the determined location of the first user, the first object entrycomprising an identifier of a first proximate object, at least onegesture action associated with the first proximate object, and at leastone program action associated with each of the gesture actions;detecting a first gesture performed by the first user with at least onesensor of the mobile device; matching the detected first gesture to atleast one gesture action in the first object entry; and executing the atleast one program action associated with the matched at least onegesture action.

In one embodiment, there is a system comprising a processor and anon-transitory storage medium storing instructions operative, whenexecuted on the processor, to perform functions including: initiatingdetection mode of a mobile device to await detecting of a tactileinteraction of a first user with a first object; detecting movements andvibrations caused by the first user interacting with the first object ina natural environment, with at least one sensor of the mobile device;and responsive to a determination that detected movements and vibrationsindicate a particular tactile interaction has occurred, initiating afirst program action associated with said tactile interaction in amemory of the mobile device.

Exemplary embodiments disclosed herein are implemented using one or morewired and/or wireless network nodes, such as a wireless transmit/receiveunit (WTRU) or other network entity.

FIG. 7 is a system diagram of an exemplary WTRU 102, which may beemployed as a mobile device (including wearable devices) in embodimentsdescribed herein. As shown in FIG. 7, the WTRU 102 may include aprocessor 118, a communication interface 119 including a transceiver120, a transmit/receive element 122, a speaker/microphone 124, a keypad126, a display/touchpad 128, a non-removable memory 130, a removablememory 132, a power source 134, a global positioning system (GPS)chipset 136, and sensors 138. It will be appreciated that the WTRU 102may include any sub-combination of the foregoing elements whileremaining consistent with an embodiment.

The processor 118 may be a general purpose processor, a special purposeprocessor, a conventional processor, a digital signal processor (DSP), aplurality of microprocessors, one or more microprocessors in associationwith a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Array (FPGAs)circuits, any other type of integrated circuit (IC), a state machine,and the like. The processor 118 may perform signal coding, dataprocessing, power control, input/output processing, and/or any otherfunctionality that enables the WTRU 102 to operate in a wirelessenvironment. The processor 118 may be coupled to the transceiver 120,which may be coupled to the transmit/receive element 122. While FIG. 7depicts the processor 118 and the transceiver 120 as separatecomponents, it will be appreciated that the processor 118 and thetransceiver 120 may be integrated together in an electronic package orchip.

The transmit/receive element 122 may be configured to transmit signalsto, or receive signals from, a base station over the air interface 116.For example, in one embodiment, the transmit/receive element 122 may bean antenna configured to transmit and/or receive RF signals. In anotherembodiment, the transmit/receive element 122 may be an emitter/detectorconfigured to transmit and/or receive IR, UV, or visible light signals,as examples. In yet another embodiment, the transmit/receive element 122may be configured to transmit and receive both RF and light signals. Itwill be appreciated that the transmit/receive element 122 may beconfigured to transmit and/or receive any combination of wirelesssignals.

In addition, although the transmit/receive element 122 is depicted inFIG. 7 as a single element, the WTRU 102 may include any number oftransmit/receive elements 122. More specifically, the WTRU 102 mayemploy MIMO technology. Thus, in one embodiment, the WTRU 102 mayinclude two or more transmit/receive elements 122 (e.g., multipleantennas) for transmitting and receiving wireless signals over the airinterface 116.

The transceiver 120 may be configured to modulate the signals that areto be transmitted by the transmit/receive element 122 and to demodulatethe signals that are received by the transmit/receive element 122. Asnoted above, the WTRU 102 may have multi-mode capabilities. Thus, thetransceiver 120 may include multiple transceivers for enabling the WTRU102 to communicate via multiple RATs, such as UTRA and IEEE 802.11, asexamples.

The processor 118 of the WTRU 102 may be coupled to, and may receiveuser input data from, the speaker/microphone 124, the keypad 126, and/orthe display/touchpad 128 (e.g., a liquid crystal display (LCD) displayunit or organic light-emitting diode (OLED) display unit). The processor118 may also output user data to the speaker/microphone 124, the keypad126, and/or the display/touchpad 128. In addition, the processor 118 mayaccess information from, and store data in, any type of suitable memory,such as the non-removable memory 130 and/or the removable memory 132.The non-removable memory 130 may include random-access memory (RAM),read-only memory (ROM), a hard disk, or any other type of memory storagedevice. The removable memory 132 may include a subscriber identitymodule (SIM) card, a memory stick, a secure digital (SD) memory card,and the like. In other embodiments, the processor 118 may accessinformation from, and store data in, memory that is not physicallylocated on the WTRU 102, such as on a server or a home computer (notshown).

The processor 118 may receive power from the power source 134, and maybe configured to distribute and/or control the power to the othercomponents in the WTRU 102. The power source 134 may be any suitabledevice for powering the WTRU 102. As examples, the power source 134 mayinclude one or more dry cell batteries (e.g., nickel-cadmium (NiCd),nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion),and the like), solar cells, fuel cells, and the like.

The processor 118 may also be coupled to the GPS chipset 136, which maybe configured to provide location information (e.g., longitude andlatitude) regarding the current location of the WTRU 102. In additionto, or in lieu of, the information from the GPS chipset 136, the WTRU102 may receive location information over the air interface 116 from abase station and/or determine its location based on the timing of thesignals being received from two or more nearby base stations. It will beappreciated that the WTRU 102 may acquire location information by way ofany suitable location-determination method while remaining consistentwith an embodiment.

The processor 118 may further be coupled to other peripherals 138, whichmay include one or more software and/or hardware modules that provideadditional features, functionality and/or wired or wirelessconnectivity. For example, the peripherals 138 may include sensors suchas an accelerometer, an e-compass, a satellite transceiver, a digitalcamera (for photographs or video), a universal serial bus (USB) port, avibration device, a haptic device or a haptic output device, atelevision transceiver, a hands free headset, a Bluetooth® module, afrequency modulated (FM) radio unit, a digital music player, a mediaplayer, a video game player module, an Internet browser, and the like.

FIG. 8 depicts an exemplary network entity 190 that may be used inembodiments of the present disclosure. As depicted in FIG. 8, networkentity 190 includes a communication interface 192, a processor 194, andnon-transitory data storage 196, all of which are communicatively linkedby a bus, network, or other communication path 198.

Communication interface 192 may include one or more wired communicationinterfaces and/or one or more wireless-communication interfaces. Withrespect to wired communication, communication interface 192 may includeone or more interfaces such as Ethernet interfaces, as an example. Withrespect to wireless communication, communication interface 192 mayinclude components such as one or more antennae, one or moretransceivers/chipsets designed and configured for one or more types ofwireless (e.g., LTE) communication, and/or any other components deemedsuitable by those of skill in the relevant art. And further with respectto wireless communication, communication interface 192 may be equippedat a scale and with a configuration appropriate for acting on thenetwork side—as opposed to the client side—of wireless communications(e.g., LTE communications, Wi Fi communications, and the like). Thus,communication interface 192 may include the appropriate equipment andcircuitry (perhaps including multiple transceivers) for serving multiplemobile stations, UEs, or other access terminals in a coverage area.

Processor 194 may include one or more processors of any type deemedsuitable by those of skill in the relevant art, some examples includinga general-purpose microprocessor and a dedicated DSP.

Data storage 196 may take the form of any non-transitorycomputer-readable medium or combination of such media, some examplesincluding flash memory, read-only memory (ROM), and random-access memory(RAM) to name but a few, as any one or more types of non-transitory datastorage deemed suitable by those of skill in the relevant art could beused. As depicted in FIG. 8, data storage 196 contains programinstructions 197 executable by processor 194 for carrying out variouscombinations of the various network-entity functions described herein.

Although features and elements are described above in particularcombinations, one of ordinary skill in the art will appreciate that eachfeature or element can be used alone or in any combination with theother features and elements. In addition, the methods described hereinmay be implemented in a computer program, software, or firmwareincorporated in a computer-readable medium for execution by a computeror processor. Examples of computer-readable storage media include, butare not limited to, a read only memory (ROM), a random access memory(RAM), a register, cache memory, semiconductor memory devices, magneticmedia such as internal hard disks and removable disks, magneto-opticalmedia, and optical media such as CD-ROM disks, and digital versatiledisks (DVDs). A processor in association with software may be used toimplement a radio frequency transceiver for use in a WTRU, UE, terminal,base station, RNC, or any host computer.

What is claimed:
 1. A method comprising: based on a user location,retrieving for each of a plurality of real-world objects (i) associatedinformation identifying a gesture associated with the respective objectand (ii) associated information identifying a program action;determining whether the user is proximate to at least a first object ofthe plurality of objects; after retrieving the information identifyingthe gesture associated with the first object, operating a sensor todetect a gesture of the user; and in response to a determination thatthe user is proximate to the first object and that the gesture of theuser corresponds to a first gesture associated with the first object,performing the program action associated with the first object.
 2. Themethod of claim 1, wherein determining whether the user is proximate tothe first object comprises determining whether the first object iswithin reach of the user.
 3. The method of claim 1, wherein theoperating of the sensor is performed in response to a determination thatthe user is proximate to at least one of the plurality of objects. 4.The method of claim 1, wherein the first gesture is a gesture performedagainst a surface of the first object.
 5. The method of claim 1, whereinthe information identifying a first gesture comprises a vibrationprofile, and wherein determining whether the gesture of the usercorresponds to the first gesture comprises determining whether thegesture of the user matches the vibration profile.
 6. The method ofclaim 5, wherein the vibration profile is based on a texture of thefirst object.
 7. The method of claim 1, further comprising determiningwhether the user is proximate to a second object of the plurality ofobjects; and in response to a determination that the user is proximateto both the first object and the second object, disambiguating thegesture of the user to determine whether to perform the program actionassociated with the first object or a program action associated with thesecond object.
 8. The method of claim 1, wherein the informationretrieved for the first object comprises (i) associated informationidentifying a second gesture associated with the first object and (ii)associated information identifying a second program action associatedwith the first object.
 9. The method of claim 1, wherein determiningwhether the user is proximate to the first object comprises determiningwhether the user is closer to the first object than to a second objectof the plurality of objects.
 10. The method of claim 1, wherein thesensor is a smart watch or a smart ring.
 11. An apparatus comprising aprocessor and a non-transitory storage medium storing instructionsoperative, when executed on the processor, to perform functionsincluding: based on a user location, retrieving for each of a pluralityof real-world objects (i) associated information identifying a gestureassociated with the respective object and (ii) associated informationidentifying a program action; determining whether the user is proximateto at least a first object of the plurality of objects; after retrievingthe information identifying the gesture associated with the firstobject, operating a sensor to detect a gesture of the user; and inresponse to a determination that the user is proximate to the firstobject and that the gesture of the user corresponds to a first gestureassociated with the first object, performing the program actionassociated with the first object.
 12. The apparatus of claim 11, whereindetermining whether the user is proximate to the first object comprisesdetermining whether the first object is within reach of the user. 13.The apparatus of claim 11, wherein the operating of the sensor isperformed in response to a determination that the user is proximate toat least one of the plurality of objects.
 14. The apparatus of claim 11,wherein the first gesture is a gesture performed against a surface ofthe first object.
 15. The apparatus of claim 11, wherein the informationidentifying a first gesture comprises a vibration profile, and whereindetermining whether the gesture of the user corresponds to the firstgesture comprises determining whether the gesture of the user matchesthe vibration profile.
 16. The apparatus of claim 15, wherein thevibration profile is based on a texture of the first object.
 17. Theapparatus of claim 11, wherein the functions further include determiningwhether the user is proximate to a second object of the plurality ofobjects; and in response to a determination that the user is proximateto both the first object and the second object, disambiguating thegesture of the user to determine whether to perform the program actionassociated with the first object or a program action associated with thesecond object.
 18. The apparatus of claim 11, wherein the informationretrieved for the first object comprises (i) associated informationidentifying a second gesture associated with the first object and (ii)associated information identifying a second program action associatedwith the first object.
 19. The apparatus of claim 11, whereindetermining whether the user is proximate to the first object comprisesdetermining whether the user is closer to the first object than to asecond object of the plurality of objects.
 20. The apparatus of claim11, wherein the sensor is a smart watch or a smart ring.